Skip to content

[Improvement](external) Increase scanner concurrency #58073

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Nov 27, 2025
Merged

[Improvement](external) Increase scanner concurrency #58073

merged 6 commits into from
Nov 27, 2025

Conversation

@Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Nov 17, 2025

What problem does this PR solve?

The concurrency of scanners are controlled by some variables which is not suitable for different cases. For external tables, we need more scanners than the internal tables. In this PR, the concurrency variable is splitted from the origin one.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Nov 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Nov 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 17, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.56% (1683/2089)
Line Coverage 66.71% (29696/44518)
Region Coverage 67.27% (14819/22028)
Branch Coverage 57.54% (7886/13706)

@doris-robot
Copy link

doris-robot commented Nov 17, 2025

TPC-H: Total hot run time: 34675 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e0bd88b8407116b6be9c54135c09920f79b3756, data reload: false

------ Round 1 ----------------------------------
q1	17471	5677	5249	5249
q2	2077	313	207	207
q3	10157	1275	714	714
q4	10223	914	383	383
q5	7503	2445	2098	2098
q6	192	178	139	139
q7	936	768	630	630
q8	9350	1386	1278	1278
q9	7419	5404	5391	5391
q10	6925	2272	1782	1782
q11	503	300	285	285
q12	339	378	233	233
q13	17800	3725	3028	3028
q14	233	237	222	222
q15	564	517	520	517
q16	1027	1005	929	929
q17	625	885	354	354
q18	7744	7111	7167	7111
q19	1099	971	583	583
q20	370	361	247	247
q21	3896	2490	2305	2305
q22	1103	1039	990	990
Total cold run time: 107556 ms
Total hot run time: 34675 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5417	5344	5321	5321
q2	277	319	236	236
q3	2247	2724	2352	2352
q4	1392	1806	1380	1380
q5	4364	5138	5032	5032
q6	188	170	135	135
q7	2012	1986	1869	1869
q8	2662	2890	2773	2773
q9	7550	7544	7466	7466
q10	3039	3304	2811	2811
q11	593	531	522	522
q12	682	793	633	633
q13	3549	3996	3347	3347
q14	307	295	302	295
q15	533	501	517	501
q16	1064	1234	1088	1088
q17	1278	1802	1329	1329
q18	7941	7842	7666	7666
q19	837	844	1196	844
q20	1924	1985	1822	1822
q21	4817	4472	4407	4407
q22	1094	1087	1036	1036
Total cold run time: 53767 ms
Total hot run time: 52865 ms

@doris-robot
Copy link

doris-robot commented Nov 17, 2025

TPC-DS: Total hot run time: 190710 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e0bd88b8407116b6be9c54135c09920f79b3756, data reload: false

query1	1011	400	422	400
query2	6573	1761	1739	1739
query3	6758	241	229	229
query4	26544	23654	23467	23467
query5	4456	660	551	551
query6	331	256	238	238
query7	4657	489	297	297
query8	301	264	254	254
query9	8704	2946	2868	2868
query10	513	376	314	314
query11	15546	15061	14854	14854
query12	191	124	121	121
query13	1697	592	533	533
query14	10560	9604	9639	9604
query15	243	188	178	178
query16	7903	693	500	500
query17	1660	844	645	645
query18	2053	436	343	343
query19	250	220	200	200
query20	137	133	125	125
query21	216	139	116	116
query22	3963	4123	4126	4123
query23	34124	33303	32748	32748
query24	8083	2426	2403	2403
query25	623	575	490	490
query26	1230	279	177	177
query27	2690	501	374	374
query28	4362	2338	2309	2309
query29	826	673	544	544
query30	306	225	200	200
query31	887	831	769	769
query32	95	89	84	84
query33	589	409	370	370
query34	773	862	572	572
query35	850	883	769	769
query36	988	1013	909	909
query37	134	123	94	94
query38	3602	3581	3594	3581
query39	1501	1439	1414	1414
query40	239	138	148	138
query41	65	59	61	59
query42	140	135	123	123
query43	492	501	458	458
query44	1292	806	814	806
query45	191	184	178	178
query46	879	991	649	649
query47	1759	1787	1713	1713
query48	413	424	338	338
query49	772	487	437	437
query50	662	689	418	418
query51	3952	4022	3930	3930
query52	127	123	117	117
query53	246	274	201	201
query54	330	315	324	315
query55	92	96	92	92
query56	347	370	359	359
query57	1172	1178	1125	1125
query58	322	297	342	297
query59	2898	2960	2925	2925
query60	374	382	351	351
query61	164	159	152	152
query62	764	704	660	660
query63	249	214	200	200
query64	4443	1164	883	883
query65	4079	3967	3953	3953
query66	1081	442	353	353
query67	15596	15305	14876	14876
query68	8296	982	660	660
query69	488	349	311	311
query70	1268	1179	1204	1179
query71	478	376	350	350
query72	6002	5144	5383	5144
query73	676	750	383	383
query74	9068	8836	8748	8748
query75	3904	3288	2820	2820
query76	3749	1172	791	791
query77	821	448	346	346
query78	9706	9743	8980	8980
query79	3014	854	638	638
query80	760	611	536	536
query81	513	270	227	227
query82	373	171	137	137
query83	301	268	250	250
query84	302	115	102	102
query85	917	479	439	439
query86	390	296	284	284
query87	3786	3762	3693	3693
query88	3608	2313	2284	2284
query89	400	340	302	302
query90	1899	255	235	235
query91	183	164	137	137
query92	80	77	72	72
query93	2078	1001	685	685
query94	688	436	333	333
query95	404	335	344	335
query96	495	593	289	289
query97	2971	2969	2898	2898
query98	266	225	230	225
query99	1505	1411	1285	1285
Total cold run time: 278459 ms
Total hot run time: 190710 ms

@doris-robot
Copy link

doris-robot commented Nov 17, 2025

ClickBench: Total hot run time: 28.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e0bd88b8407116b6be9c54135c09920f79b3756, data reload: false

query1	0.05	0.04	0.04
query2	0.14	0.07	0.07
query3	0.32	0.08	0.08
query4	1.61	0.10	0.09
query5	0.27	0.27	0.25
query6	1.19	0.64	0.64
query7	0.03	0.03	0.02
query8	0.08	0.06	0.06
query9	0.60	0.53	0.52
query10	0.58	0.58	0.56
query11	0.25	0.14	0.14
query12	0.26	0.15	0.15
query13	0.63	0.64	0.61
query14	1.01	1.01	1.00
query15	0.86	0.82	0.82
query16	0.39	0.41	0.40
query17	1.08	1.06	0.98
query18	0.24	0.23	0.22
query19	1.92	1.91	1.84
query20	0.02	0.02	0.01
query21	15.42	0.29	0.24
query22	5.00	0.12	0.10
query23	15.39	0.36	0.23
query24	2.28	0.45	0.32
query25	0.10	0.10	0.10
query26	0.20	0.18	0.18
query27	0.10	0.09	0.09
query28	3.70	1.26	1.10
query29	12.55	4.03	3.34
query30	0.35	0.13	0.12
query31	2.82	0.64	0.44
query32	3.28	0.60	0.51
query33	3.04	3.06	3.08
query34	16.09	5.14	4.59
query35	4.59	4.61	4.61
query36	0.63	0.51	0.49
query37	0.24	0.09	0.09
query38	0.20	0.05	0.06
query39	0.06	0.04	0.05
query40	0.22	0.17	0.16
query41	0.12	0.08	0.06
query42	0.07	0.05	0.05
query43	0.07	0.05	0.06
Total cold run time: 98.05 s
Total hot run time: 28.52 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 17, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Nov 17, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 17, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.56% (1683/2089)
Line Coverage 66.68% (29684/44518)
Region Coverage 67.22% (14808/22028)
Branch Coverage 57.52% (7884/13706)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 17, 2025

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Nov 17, 2025

TPC-H: Total hot run time: 36006 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 22249e2f89b2266965806f7444efdeb3e16ab171, data reload: false

------ Round 1 ----------------------------------
q1	17401	5569	5188	5188
q2	2010	307	206	206
q3	10161	1242	713	713
q4	10225	897	348	348
q5	7498	2443	2354	2354
q6	187	176	137	137
q7	939	761	625	625
q8	9355	1426	1319	1319
q9	7375	5294	5418	5294
q10	6912	2263	1789	1789
q11	501	292	272	272
q12	331	367	220	220
q13	17784	3627	3016	3016
q14	241	236	208	208
q15	569	529	499	499
q16	987	1026	928	928
q17	593	873	342	342
q18	8234	7529	7665	7529
q19	1123	977	597	597
q20	380	369	256	256
q21	4242	3384	3099	3099
q22	1200	1081	1067	1067
Total cold run time: 108248 ms
Total hot run time: 36006 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5489	5571	5486	5486
q2	245	326	230	230
q3	2489	2810	2467	2467
q4	1478	1879	1461	1461
q5	4571	4948	5022	4948
q6	177	160	126	126
q7	2014	1963	2009	1963
q8	2543	2828	2623	2623
q9	7402	7527	7230	7230
q10	2874	3167	2623	2623
q11	573	504	497	497
q12	625	709	574	574
q13	3267	3651	3001	3001
q14	272	273	253	253
q15	526	503	477	477
q16	1021	1043	979	979
q17	1180	1694	1219	1219
q18	7288	7261	7141	7141
q19	760	988	1060	988
q20	1909	1940	1823	1823
q21	4758	4399	4297	4297
q22	1098	1039	995	995
Total cold run time: 52559 ms
Total hot run time: 51401 ms

@doris-robot
Copy link

doris-robot commented Nov 17, 2025

TPC-DS: Total hot run time: 189875 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 22249e2f89b2266965806f7444efdeb3e16ab171, data reload: false

query1	1054	406	426	406
query2	6543	1727	1749	1727
query3	6755	227	220	220
query4	26577	23441	23226	23226
query5	4410	658	531	531
query6	330	243	234	234
query7	4647	485	299	299
query8	305	269	242	242
query9	8674	2910	2878	2878
query10	517	363	317	317
query11	15621	15136	14834	14834
query12	181	125	116	116
query13	1664	580	462	462
query14	10564	9108	9167	9108
query15	198	185	173	173
query16	7797	640	505	505
query17	1376	742	601	601
query18	2039	431	315	315
query19	244	222	237	222
query20	135	133	123	123
query21	211	130	118	118
query22	4186	4422	4280	4280
query23	34118	32974	32894	32894
query24	8050	2261	2379	2261
query25	588	532	465	465
query26	1241	279	160	160
query27	2720	496	361	361
query28	4414	2303	2279	2279
query29	786	626	537	537
query30	301	227	199	199
query31	881	787	711	711
query32	94	80	80	80
query33	566	384	358	358
query34	763	851	527	527
query35	830	858	756	756
query36	983	1023	881	881
query37	124	119	99	99
query38	3582	3486	3507	3486
query39	1472	1421	1417	1417
query40	230	135	124	124
query41	64	61	60	60
query42	138	121	118	118
query43	474	496	447	447
query44	1281	806	799	799
query45	192	195	185	185
query46	880	966	661	661
query47	1804	1853	1776	1776
query48	413	431	355	355
query49	793	530	450	450
query50	655	676	411	411
query51	3890	4020	3912	3912
query52	124	119	110	110
query53	237	276	207	207
query54	324	312	304	304
query55	91	94	88	88
query56	337	351	378	351
query57	1169	1193	1126	1126
query58	315	305	304	304
query59	2863	2957	2909	2909
query60	394	393	363	363
query61	193	185	195	185
query62	766	740	687	687
query63	236	202	199	199
query64	4592	1314	988	988
query65	4033	3917	3943	3917
query66	1093	455	373	373
query67	15446	15152	15088	15088
query68	8114	949	662	662
query69	509	347	314	314
query70	1277	1236	1229	1229
query71	440	378	356	356
query72	6395	5123	5315	5123
query73	657	760	370	370
query74	8899	9124	8779	8779
query75	3299	3262	2857	2857
query76	3288	1119	748	748
query77	548	411	349	349
query78	9755	9852	8978	8978
query79	2845	820	612	612
query80	680	602	526	526
query81	503	263	228	228
query82	438	169	139	139
query83	298	271	248	248
query84	306	111	101	101
query85	927	484	444	444
query86	378	303	294	294
query87	3694	3702	3622	3622
query88	3674	2278	2275	2275
query89	396	345	303	303
query90	2000	236	223	223
query91	169	164	137	137
query92	85	78	71	71
query93	2101	954	684	684
query94	726	432	327	327
query95	403	345	332	332
query96	482	593	295	295
query97	2941	2953	2854	2854
query98	261	220	218	218
query99	1446	1385	1272	1272
Total cold run time: 276502 ms
Total hot run time: 189875 ms

@doris-robot
Copy link

doris-robot commented Nov 17, 2025

ClickBench: Total hot run time: 28.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 22249e2f89b2266965806f7444efdeb3e16ab171, data reload: false

query1	0.05	0.04	0.05
query2	0.13	0.07	0.06
query3	0.32	0.07	0.08
query4	1.60	0.08	0.09
query5	0.26	0.26	0.27
query6	1.18	0.64	0.64
query7	0.03	0.03	0.03
query8	0.08	0.06	0.07
query9	0.58	0.51	0.51
query10	0.58	0.55	0.56
query11	0.24	0.13	0.14
query12	0.26	0.15	0.15
query13	0.63	0.62	0.62
query14	1.00	0.98	0.99
query15	0.85	0.81	0.80
query16	0.40	0.38	0.39
query17	1.04	1.04	1.10
query18	0.23	0.22	0.22
query19	1.87	1.81	1.79
query20	0.02	0.02	0.01
query21	15.40	0.27	0.24
query22	4.98	0.10	0.09
query23	15.38	0.37	0.22
query24	2.18	0.46	0.35
query25	0.09	0.09	0.09
query26	0.18	0.18	0.17
query27	0.09	0.09	0.08
query28	4.01	1.26	1.08
query29	12.54	3.96	3.26
query30	0.35	0.12	0.12
query31	2.82	0.63	0.42
query32	3.23	0.58	0.50
query33	3.03	3.12	3.08
query34	16.27	5.18	4.59
query35	4.58	4.62	4.62
query36	0.61	0.51	0.48
query37	0.24	0.09	0.08
query38	0.19	0.06	0.06
query39	0.06	0.05	0.04
query40	0.19	0.18	0.16
query41	0.12	0.06	0.07
query42	0.07	0.06	0.06
query43	0.07	0.06	0.05
Total cold run time: 98.03 s
Total hot run time: 28.32 s

@doris-robot
Copy link

doris-robot commented Nov 17, 2025

BE UT Coverage Report

Increment line coverage 20.51% (8/39) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.74% (18274/34646)
Line Coverage 38.12% (166087/435738)
Region Coverage 33.08% (129138/390377)
Branch Coverage 33.83% (55395/163759)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 18, 2025

BE Regression && UT Coverage Report

Increment line coverage 97.22% (35/36) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.51% (24347/34046)
Line Coverage 57.98% (253031/436409)
Region Coverage 53.17% (210489/395881)
Branch Coverage 54.57% (89951/164834)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 18, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

morningman
morningman reviewed Nov 24, 2025
View reviewed changes
be/src/pipeline/exec/file_scan_operator.cpp Outdated
#include "common/compile_check_begin.h"

int FileScanLocalState::max_scanners_concurrency(RuntimeState* state) {
return (state->max_file_scanners_concurrency()
Copy link
Contributor

@morningman morningman Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add some comment for this logic

@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Nov 24, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 24, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.86% (1707/2111)
Line Coverage 66.85% (29887/44708)
Region Coverage 67.38% (14900/22114)
Branch Coverage 57.57% (7926/13768)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 24, 2025

FE UT Coverage Report

Increment line coverage 66.67% (4/6) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Nov 24, 2025

TPC-H: Total hot run time: 34278 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6d4a5a626312a4f4c9a77a0ad5c412dca03620b0, data reload: false

------ Round 1 ----------------------------------
q1	17594	5044	4891	4891
q2	2029	314	199	199
q3	10207	1295	713	713
q4	10239	960	364	364
q5	7554	2531	2263	2263
q6	190	173	139	139
q7	905	773	629	629
q8	9290	1336	1020	1020
q9	7329	5357	5377	5357
q10	6861	2231	1825	1825
q11	500	306	294	294
q12	359	370	228	228
q13	17796	3739	3050	3050
q14	243	241	218	218
q15	572	511	527	511
q16	1030	1011	934	934
q17	593	747	498	498
q18	7378	7181	7144	7144
q19	1102	953	563	563
q20	345	351	228	228
q21	3582	2605	2292	2292
q22	1012	968	918	918
Total cold run time: 106710 ms
Total hot run time: 34278 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4943	4892	4904	4892
q2	332	393	320	320
q3	2133	2695	2298	2298
q4	1305	1758	1293	1293
q5	4187	4314	4422	4314
q6	227	173	135	135
q7	2048	1940	1836	1836
q8	2664	2444	2463	2444
q9	7625	7648	7590	7590
q10	3059	3250	2830	2830
q11	586	515	504	504
q12	693	747	652	652
q13	3720	3849	3352	3352
q14	288	327	289	289
q15	528	510	512	510
q16	1062	1086	1036	1036
q17	1144	1506	1416	1416
q18	7819	7864	7513	7513
q19	766	749	761	749
q20	2157	2039	1911	1911
q21	4769	4249	4193	4193
q22	1051	988	967	967
Total cold run time: 53106 ms
Total hot run time: 51044 ms

@doris-robot
Copy link

doris-robot commented Nov 24, 2025

TPC-DS: Total hot run time: 187790 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6d4a5a626312a4f4c9a77a0ad5c412dca03620b0, data reload: false

query1	1001	423	387	387
query2	6555	1679	1666	1666
query3	6752	215	227	215
query4	25758	23675	23446	23446
query5	4929	653	487	487
query6	346	235	220	220
query7	4671	495	317	317
query8	295	248	249	248
query9	8719	2608	2640	2608
query10	513	347	287	287
query11	15625	15152	15184	15152
query12	186	122	115	115
query13	1690	570	464	464
query14	11146	9054	9009	9009
query15	203	198	172	172
query16	7597	721	504	504
query17	1180	749	618	618
query18	2020	426	319	319
query19	204	193	168	168
query20	135	127	123	123
query21	220	131	113	113
query22	4087	4322	3986	3986
query23	33936	32990	32812	32812
query24	8474	2389	2364	2364
query25	634	523	460	460
query26	1236	275	166	166
query27	2734	497	357	357
query28	4348	2187	2174	2174
query29	760	668	483	483
query30	301	221	196	196
query31	918	803	724	724
query32	82	76	73	73
query33	580	358	324	324
query34	804	856	524	524
query35	802	808	716	716
query36	910	948	897	897
query37	114	114	93	93
query38	3478	3521	3440	3440
query39	1451	1417	1394	1394
query40	225	131	128	128
query41	71	62	65	62
query42	126	112	107	107
query43	466	485	450	450
query44	1242	773	772	772
query45	186	180	171	171
query46	869	981	648	648
query47	1778	1817	1724	1724
query48	419	410	323	323
query49	777	482	398	398
query50	652	678	410	410
query51	3973	3911	3859	3859
query52	114	113	108	108
query53	244	259	198	198
query54	302	291	273	273
query55	89	91	83	83
query56	321	325	313	313
query57	1167	1187	1110	1110
query58	292	273	275	273
query59	2474	2551	2506	2506
query60	342	346	348	346
query61	160	164	167	164
query62	795	730	656	656
query63	233	195	193	193
query64	4414	1185	937	937
query65	4030	3935	3924	3924
query66	1139	454	352	352
query67	15114	15013	14881	14881
query68	4607	963	650	650
query69	503	341	304	304
query70	1271	1195	1265	1195
query71	419	328	312	312
query72	6042	5027	4974	4974
query73	662	565	354	354
query74	9295	9071	9043	9043
query75	3279	3303	2812	2812
query76	3307	1129	707	707
query77	499	409	329	329
query78	9557	9730	8828	8828
query79	1894	865	603	603
query80	723	583	525	525
query81	518	258	225	225
query82	240	166	130	130
query83	278	275	250	250
query84	265	105	89	89
query85	896	488	446	446
query86	384	308	292	292
query87	3669	3711	3572	3572
query88	2783	2274	2243	2243
query89	383	338	294	294
query90	1826	223	221	221
query91	195	163	147	147
query92	79	68	69	68
query93	1236	1009	690	690
query94	638	446	339	339
query95	418	321	323	321
query96	491	550	290	290
query97	2858	2950	2871	2871
query98	246	214	204	204
query99	1284	1405	1272	1272
Total cold run time: 268153 ms
Total hot run time: 187790 ms

@doris-robot
Copy link

doris-robot commented Nov 24, 2025

ClickBench: Total hot run time: 28.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6d4a5a626312a4f4c9a77a0ad5c412dca03620b0, data reload: false

query1	0.05	0.05	0.04
query2	0.13	0.06	0.06
query3	0.33	0.08	0.08
query4	1.61	0.09	0.09
query5	0.26	0.25	0.24
query6	1.18	0.65	0.63
query7	0.03	0.02	0.03
query8	0.07	0.06	0.06
query9	0.59	0.53	0.51
query10	0.57	0.55	0.55
query11	0.25	0.12	0.14
query12	0.25	0.14	0.14
query13	0.64	0.62	0.61
query14	1.01	0.99	1.01
query15	0.89	0.83	0.82
query16	0.39	0.39	0.40
query17	0.96	1.08	1.05
query18	0.22	0.20	0.21
query19	1.86	1.77	1.80
query20	0.02	0.01	0.01
query21	15.43	0.27	0.23
query22	4.98	0.09	0.09
query23	15.39	0.38	0.22
query24	2.34	0.45	0.31
query25	0.09	0.09	0.10
query26	0.18	0.17	0.18
query27	0.09	0.08	0.08
query28	3.55	1.26	1.09
query29	12.56	3.93	3.32
query30	0.33	0.13	0.12
query31	2.82	0.60	0.44
query32	3.23	0.58	0.50
query33	3.08	3.13	3.02
query34	16.38	5.29	4.63
query35	4.68	4.67	4.69
query36	0.62	0.52	0.48
query37	0.23	0.09	0.09
query38	0.20	0.06	0.06
query39	0.06	0.05	0.04
query40	0.20	0.18	0.16
query41	0.11	0.06	0.06
query42	0.07	0.06	0.05
query43	0.06	0.05	0.04
Total cold run time: 97.99 s
Total hot run time: 28.34 s

@doris-robot
Copy link

doris-robot commented Nov 24, 2025

BE UT Coverage Report

Increment line coverage 18.60% (8/43) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.63% (18280/34735)
Line Coverage 38.05% (166262/436947)
Region Coverage 33.00% (129229/391573)
Branch Coverage 33.80% (55461/164082)

@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Nov 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 25, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.86% (1707/2111)
Line Coverage 66.86% (29891/44708)
Region Coverage 67.41% (14906/22114)
Branch Coverage 57.57% (7926/13768)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 25, 2025

FE UT Coverage Report

Increment line coverage 66.67% (4/6) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Nov 25, 2025

TPC-H: Total hot run time: 36199 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f1aaee068839444ae7dd524bf84bcce15c4fba9, data reload: false

------ Round 1 ----------------------------------
q1	17467	5495	5139	5139
q2	2024	329	211	211
q3	10187	1225	729	729
q4	10235	904	377	377
q5	7540	2470	2156	2156
q6	194	170	136	136
q7	956	783	645	645
q8	9349	1434	1263	1263
q9	7513	5414	5402	5402
q10	6941	2288	1835	1835
q11	501	294	294	294
q12	336	365	225	225
q13	17793	3650	3061	3061
q14	233	235	209	209
q15	589	516	507	507
q16	1035	1011	961	961
q17	611	885	364	364
q18	8116	7647	7691	7647
q19	1116	988	591	591
q20	387	367	246	246
q21	4364	3852	3170	3170
q22	1180	1104	1031	1031
Total cold run time: 108667 ms
Total hot run time: 36199 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5399	5487	5373	5373
q2	329	411	340	340
q3	2514	2824	2492	2492
q4	1455	1887	1441	1441
q5	4585	5087	5005	5005
q6	173	168	126	126
q7	2060	2013	1790	1790
q8	2849	2839	2705	2705
q9	7753	7795	7444	7444
q10	3046	3201	2685	2685
q11	578	505	476	476
q12	621	730	581	581
q13	3278	3674	3095	3095
q14	271	284	259	259
q15	537	502	502	502
q16	1045	1079	1027	1027
q17	1204	1714	1230	1230
q18	7373	7270	7111	7111
q19	798	1152	1022	1022
q20	1947	2003	1836	1836
q21	4859	4550	4391	4391
q22	1139	1073	1028	1028
Total cold run time: 53813 ms
Total hot run time: 51959 ms

@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Nov 27, 2025

run buildall

@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Nov 27, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 27, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.86% (1707/2111)
Line Coverage 66.84% (29882/44708)
Region Coverage 67.38% (14901/22114)
Branch Coverage 57.59% (7929/13768)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

FE UT Coverage Report

Increment line coverage 80.00% (8/10) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-H: Total hot run time: 34513 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b87eaf23bd13dcfd21756e0823e75065d36d475b, data reload: false

------ Round 1 ----------------------------------
q1	17405	5135	4954	4954
q2	2060	320	214	214
q3	10147	1299	735	735
q4	10259	881	371	371
q5	7525	2519	2246	2246
q6	180	167	137	137
q7	949	813	635	635
q8	9347	1365	1032	1032
q9	7147	5505	5371	5371
q10	6882	2223	1828	1828
q11	529	322	296	296
q12	358	375	232	232
q13	17769	3668	3044	3044
q14	235	226	210	210
q15	591	521	509	509
q16	1050	1023	954	954
q17	606	773	506	506
q18	8026	7101	7060	7060
q19	1090	961	590	590
q20	343	342	231	231
q21	3564	3696	2396	2396
q22	1003	968	962	962
Total cold run time: 107065 ms
Total hot run time: 34513 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5155	4955	4913	4913
q2	339	397	309	309
q3	2209	2707	2277	2277
q4	1316	1756	1360	1360
q5	4267	4593	4621	4593
q6	224	175	130	130
q7	2077	2009	1823	1823
q8	2706	2516	2473	2473
q9	7546	7504	7473	7473
q10	3082	3342	2814	2814
q11	634	517	493	493
q12	660	735	595	595
q13	3593	3936	3294	3294
q14	297	328	290	290
q15	556	504	512	504
q16	1111	1132	1085	1085
q17	1166	1473	1412	1412
q18	7848	7782	7643	7643
q19	803	818	821	818
q20	2018	2041	1906	1906
q21	4879	4337	4213	4213
q22	1037	1013	981	981
Total cold run time: 53523 ms
Total hot run time: 51399 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-DS: Total hot run time: 184571 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b87eaf23bd13dcfd21756e0823e75065d36d475b, data reload: false

query1	1110	427	392	392
query2	6578	1577	1615	1577
query3	6750	222	226	222
query4	25674	23362	22880	22880
query5	4902	625	483	483
query6	343	249	226	226
query7	4642	510	309	309
query8	302	252	240	240
query9	8708	2577	2572	2572
query10	545	373	302	302
query11	15159	15203	14977	14977
query12	180	117	115	115
query13	1707	571	484	484
query14	11073	8907	8749	8749
query15	212	203	194	194
query16	7526	703	541	541
query17	1220	793	655	655
query18	2013	416	322	322
query19	207	198	171	171
query20	133	124	119	119
query21	215	147	115	115
query22	3996	3951	3877	3877
query23	33039	31988	31758	31758
query24	8410	2383	2387	2383
query25	604	518	444	444
query26	1243	283	168	168
query27	2736	487	339	339
query28	4339	2133	2086	2086
query29	831	639	491	491
query30	314	240	207	207
query31	852	732	589	589
query32	81	74	73	73
query33	600	377	335	335
query34	800	874	546	546
query35	802	855	763	763
query36	896	942	851	851
query37	122	111	92	92
query38	3356	3326	3284	3284
query39	1461	1384	1403	1384
query40	234	132	117	117
query41	65	64	63	63
query42	124	118	110	110
query43	429	460	420	420
query44	1293	759	755	755
query45	201	188	184	184
query46	874	997	645	645
query47	1692	1703	1631	1631
query48	400	423	328	328
query49	781	503	428	428
query50	659	677	410	410
query51	3885	3926	3838	3838
query52	112	113	99	99
query53	243	263	189	189
query54	309	291	278	278
query55	98	96	89	89
query56	336	335	331	331
query57	1156	1148	1090	1090
query58	295	275	269	269
query59	2389	2364	2236	2236
query60	346	344	339	339
query61	166	164	167	164
query62	799	707	681	681
query63	227	195	195	195
query64	4561	1306	1014	1014
query65	4085	3962	4023	3962
query66	1142	466	363	363
query67	15486	14995	14845	14845
query68	7399	952	628	628
query69	525	369	315	315
query70	1287	1203	1210	1203
query71	442	347	328	328
query72	6036	4863	4815	4815
query73	659	573	341	341
query74	8815	8753	8683	8683
query75	3329	3342	2865	2865
query76	3329	1126	699	699
query77	509	405	330	330
query78	9476	9642	8940	8940
query79	2276	803	593	593
query80	634	580	531	531
query81	520	272	249	249
query82	431	160	131	131
query83	260	262	244	244
query84	256	107	97	97
query85	924	487	452	452
query86	382	296	282	282
query87	3485	3474	3463	3463
query88	4125	2237	2250	2237
query89	397	330	297	297
query90	1931	222	216	216
query91	174	171	139	139
query92	89	74	69	69
query93	1975	987	663	663
query94	790	470	366	366
query95	496	404	408	404
query96	509	565	300	300
query97	2893	2981	2842	2842
query98	244	221	210	210
query99	1352	1419	1251	1251
Total cold run time: 272017 ms
Total hot run time: 184571 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

ClickBench: Total hot run time: 27.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b87eaf23bd13dcfd21756e0823e75065d36d475b, data reload: false

query1	0.05	0.05	0.04
query2	0.12	0.05	0.04
query3	0.26	0.08	0.09
query4	1.60	0.12	0.11
query5	0.28	0.25	0.25
query6	1.16	0.65	0.64
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.59	0.50	0.51
query10	0.56	0.58	0.56
query11	0.17	0.11	0.10
query12	0.16	0.12	0.11
query13	0.62	0.60	0.61
query14	0.98	1.01	1.00
query15	0.82	0.80	0.81
query16	0.41	0.41	0.42
query17	1.07	1.04	1.00
query18	0.23	0.22	0.22
query19	1.94	1.73	1.73
query20	0.02	0.02	0.02
query21	15.45	0.26	0.14
query22	4.94	0.05	0.05
query23	15.99	0.27	0.10
query24	1.21	0.66	0.18
query25	0.08	0.11	0.05
query26	0.14	0.13	0.13
query27	0.05	0.04	0.06
query28	4.07	1.21	1.02
query29	12.56	3.90	3.21
query30	0.29	0.13	0.12
query31	2.82	0.62	0.40
query32	3.24	0.54	0.46
query33	3.07	3.06	3.10
query34	16.86	5.20	4.52
query35	4.58	4.56	4.60
query36	0.67	0.50	0.49
query37	0.10	0.07	0.06
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.17	0.13	0.13
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 97.71 s
Total hot run time: 27.16 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 27, 2025

PR approved by at least one committer and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

FE Regression Coverage Report

Increment line coverage 0.00% (0/10) 🎉
Increment coverage report
Complete coverage report

kaka11chen
kaka11chen approved these changes Nov 27, 2025
View reviewed changes
Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

skip check_coverage

@yiguolei yiguolei merged commit 1bb4669 into apache:master Nov 27, 2025
28 of 30 checks passed
@morningman morningman mentioned this pull request Dec 1, 2025
Merged
BiteTheDDDDt pushed a commit that referenced this pull request Dec 2, 2025
@Chen768959
[fix](ann index) Update ivf_nprobe struct_id (#58588)
a683101 
build error.
Update ivf_nprobe struct_id

introduced from #58130 and #58073
airborne12 pushed a commit to airborne12/apache-doris that referenced this pull request Dec 2, 2025
@Chen768959 @airborne12
[fix](ann index) Update ivf_nprobe struct_id (apache#58588)
85ba848 
build error.
Update ivf_nprobe struct_id

introduced from apache#58130 and apache#58073
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Dec 3, 2025
@Gabriel39
[Improvement](external) Increase scanner concurrency (apache#58073)
72e33b8 
The concurrency of scanners are controlled by some variables which is
not suitable for different cases. For external tables, we need more
scanners than the internal tables. In this PR, the concurrency variable
is splitted from the origin one.

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
yiguolei pushed a commit that referenced this pull request Dec 3, 2025
@Gabriel39
[Improvement](external) Increase scanner concurrency (#58073) (#58678)
45a97ef 
The concurrency of scanners are controlled by some variables which is
not suitable for different cases. For external tables, we need more
scanners than the internal tables. In this PR, the concurrency variable
is splitted from the origin one.

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman left review comments

@kaka11chen kaka11chen kaka11chen approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.x dev/4.0.x-conflict reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Gabriel39 @Thearas @doris-robot @hello-stephen @kaka11chen @morningman @yiguolei